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I.  INTRODUCTION 


A.  BACKGROUND 

Naval  Air  Systems  Command  (NAVAIR)  is  the  primary  acquisition  agency  for 
Naval  aircraft,  airborne  weapons,  and  aviation  systems.  Currently,  NAVAIR  has  very 
formal  processes,  procedures,  and  doctrine  on  how  the  systems  engineering  process  is  to 
be  perfonned  throughout  the  acquisition  process.  A  good  example  is  the  Systems 
Engineering  Technical  Review  (SETR)  process,  which  requires  programs  to  go  through 
formal  review  steps  to  ensure  that  the  systems  engineering  process  is  being  conducted  in 
a  satisfactory  manner.  The  Test  and  Evaluation  (T&E)  community,  however,  lacks 
similar  direction.  There  is  very  limited  guidance  as  to  how  T&E  should  be  integrated 
into  each  stage  of  the  acquisition  process.  As  a  result,  the  T&E  community  is  often 
brought  in  on  an  “as-needed”  basis  whenever  the  program  manager  or  systems  engineer 
recognizes  that  T&E  help  is  needed.  This  in  turn  causes  the  vast  majority  of  T&E 
participation  on  programs  to  occur  post-Milestone  B.  T&E  participation  and  insight 
during  the  earlier  stages  of  the  program  may  be  very  beneficial  but  rarely  occur  in  an 
effective  manner  due  to  lack  of  guidance  in  the  T&E  community. 

The  situation  described  has  led  the  NAVAIR  T&E  community  to  begin  an 
initiative  to  develop  better  guidance  on  how  the  community  can  integrate  better  with  the 
other  competencies  (engineering,  systems  engineering,  program  management,  logistics, 
etc.)  and  acquisition  policies/procedures  during  all  phases  of  the  acquisition  process, 
especially  prior  to  Milestone  B.  This  thesis  will  support  this  initiative  by  identifying 
areas  of  weakness  integrating  T&E  into  the  acquisition  process  and  by  seeking 
opportunities  for  improvement. 
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B.  PURPOSE 

The  purpose  of  this  thesis  is  to  understand  shortcomings  experienced  by  elements 
of  the  Naval  Aviation  acquisition  community  integrating  T&E  processes  and  procedures 
into  all  phases  of  the  system  acquisition  process.  Further,  this  thesis  makes 
recommendations  to  overcome  these  shortcomings. 

C.  RESEARCH  QUESTIONS 

1 .  What  guidance  is  currently  provided  by  Department  of  Defense  (DoD), 
Department  of  Navy  (DoN),  and  Naval  Air  Systems  Command 
(NAVAIR)  regarding  integrating  T&E  into  the  systems  acquisition 
process? 

2.  What  areas  of  weakness  exist  regarding  integrating  T&E  into  the  systems 
acquisition  process? 

3.  What  improvements  can  be  made  to  policies,  procedures,  and  guidance  to 
better  integrate  T&E  into  the  systems  acquisition  process? 

D.  BENEFITS  OF  STUDY 

This  thesis  provides  a  basis  of  knowledge  that  can  be  used  by  NAVAIR  and 
leveraged  by  other  DoD  T&E  activities,  improving  the  integration  of  T&E  into  the 
systems  acquisition  process. 

E.  SCOPE 

This  thesis  focuses  on  processes  for  integrating  T&E  into  the  systems  acquisition 
process  as  derived  from  analysis  of  T&E  policy,  guidance,  and  input  from  senior  Naval 
Aviation  T&E  managers,  systems  engineers,  and  program  managers.  Much  of  the 
analysis  is  dependent  on  interview  responses. 
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F.  METHODOLOGY 

1.  Conducted  literature  review  of  T&E  and  systems  acquisition  regulations, 
policy,  procedures,  guidance,  other  pertinent  T&E  related  material. 

2.  Analyzed  regulations,  policy,  procedures,  and  guidance  to  determine 
shortcomings. 

3.  Developed  appropriate  interview  questions. 

4.  Interviewed  senior  Naval  Aviation  T&E  managers,  engineers,  systems 
engineers,  and  program  managers  and  analyzed  responses  to  detennine  current 
shortcomings. 

5.  Developed  recommendations  for  improving  guidance  for  integrating  T&E  into 
the  systems  acquisition  process. 
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II.  T&E  IN  THE  SYSTEMS  ENGINEERING  PROCESS 


A.  INTRODUCTION 

Before  looking  at  how  T&E  is  currently  integrated  into  the  Naval  Aviation 
acquisition  process  through  policy  and  guidance,  this  chapter  will  examine: 

•  What  is  the  systems  engineering  process? 

•  What  is  T&E’s  role  in  the  systems  engineering  process? 

While  various  resources  will  be  used  in  this  exploration  of  the  systems 
engineering  process,  including  those  from  government,  industry  and  academia,  this 
chapter  will  focus  on  reviewing  the  systems  engineering  process  as  given  in  DoD 
resources,  as  this  should  form  the  basis  for  DoD  T&E  guidance.  By  investigating  how 
T&E  is  “supposed”  to  be  integrated  into  the  systems  engineering  process,  later  chapters 
will  then  identify  and  analyze  any  shortcomings  of  the  current  DoD,  DoN,  and  NAVAIR 
policies  and  guidance. 

B.  THE  SYSTEMS  ENGINEERING  PROCESS 

Systems  engineering  has  been  defined  in  many  ways,  including: 

Systems  engineering  is  an  interdisciplinary  approach  encompassing  the 
entire  technical  effort  to  evolve  and  verify  an  integrated  and  total  lifecycle 
balanced  set  of  system,  people,  and  process  solutions  that  satisfy  customer 
needs.  Systems  engineering  is  the  integrating  mechanism  across  the 
technical  efforts  related  to  the  development,  manufacturing,  verification, 
deployment,  operations,  support,  disposal  of,  and  user  training  for  systems 
and  their  life  cycle  processes.  System  engineering  develops  technical 
information  to  support  the  program  management  decision-making  process. 

[1] 


The  function  of  systems  engineering  is  to  guide  the  engineering  of 
complex  systems.  [2] 
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An  interdisciplinary  approach  and  means  to  enable  the  realization  of 
successful  systems.  [3] 


Systems  engineering  is  an  interdisciplinary  approach  to  evolve  and  verify 
an  integrated  and  optimally  balanced  set  of  product  and  process  designs 
that  satisfy  user  needs  and  provide  information  for  management  decision 
making.  [4] 

Through  the  practice  of  systems  engineering,  a  systematic  methodology  has  been 
developed  for  transfonning  a  set  of  requirements  into  an  operational  system  that  meets 
those  requirements.  The  systems  engineering  process  may  be  defined  as: 

The  systems  engineering  process  is  the  iterative  logical  sequence  of 
analysis,  design,  test,  and  decision  activities  that  transforms  an  operational 
need  into  the  descriptions  required  for  production  and  fielding  of  all 
operational  and  support  system  elements.  [4] 

While  the  systems  engineering  process  is  defined  and  visualized  using  many 
different  models,  this  thesis  will  focus  on  how  it  is  defined  and  presented  in  the  Defense 
Acquisition  Guidebook  (DAG)  [1]  as  this  provides  official  guidance  on  how  the  systems 
engineering  process  should  be  accomplished  in  DoD  programs.  The  DAG  divides  the 
systems  engineering  process  into  eight  technical  processes  (three  design  processes  and 
five  realization  process)  and  eight  technical  management  processes.  These  processes  are 
illustrated  in  Figure  1  and  further  discussed  below. 
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FIGURE  1 .  SYSTEMS  ENGINEERING  PROCESS  MODEL  [FROM  5] 

The  system  developer  uses  technical  processes  to  transform  a  system  from  an 
identified  need  all  the  way  to  a  fielded  system.  The  first  technical  process  is 
requirements  development. 

The  requirements  development  process  translates  inputs  from  the  relevant 
stakeholders  into  technical  requirements.  Working  with  the  user,  the  system  developer 
translates  the  user’s  needs  into  performance  parameter  objectives  and  thresholds, 
affordability  constraints,  schedule  constraints,  and  technical  constraints.  Requirements 
development  is  an  iterative  process  with  the  goal  of  outputting  requirements  with  the 
proper  balance  between  performance  and  affordable  cost. 

Next,  each  system  level  function  is  analyzed  using  logical  analysis.  This  is  a 
recursive  process,  allocating  requirements  from  higher  levels  to  lower  levels,  providing 
traceable  requirements.  Logical  analysis  obtains  sets  of  logical  solutions  to  better 
understand  the  defined  requirements  and  the  relationships  among  them.  Once  these 
solution  sets  are  formed,  performance  parameters  and  constraints  can  be  allocated. 
Technical  requirements  can  then  be  defined  from  the  allocated  performance  parameters. 
The  product  of  this  step  is  a  description  of  the  system  in  functional  terms. 
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Through  the  design  solution  process,  the  system  developer  then  translates  the 
functional  architecture  developed  during  logical  analysis  into  the  physical  hardware  and 
software  component  design.  Each  design  element  must  be  consistent  with  the  functional 
analysis,  performing  the  function  that  is  intended  to  the  level  of  performance  required. 
The  output  of  this  process  is  the  physical  architecture  that  forms  the  basis  for  design 
definition  documentation. 

Next,  implementation  produces  the  lowest  level  system  elements  in  the  system 
hierarchy.  This  may  be  accomplished  by  making,  buying,  or  reusing  the  system 
elements.  This  process  may  also  include  developing  a  manufacturing  process  if  the 
implementation  involves  production. 

During  integration,  the  lower-level  system  elements  are  integrated  into  higher- 
level  system  elements  in  the  physical  architecture.  This  process  assembles  the  full 
system  from  its  components. 

The  verification  process  confirms  that  the  system  meets  the  design-to  or  built-to 
specifications.  Through  developmental  testing,  verification  answers  the  question,  “Did 
you  build  it  right?” 

The  validation  process  evaluates  the  performance  of  the  system  within  its 
intended  operational  environment  with  intended  operational  users.  Through  operational 
testing,  validation  answers  the  question,  “Did  you  build  the  right  thing?” 

Finally,  the  transition  process  moves  the  system  to  the  use  of  the  end-term  users. 

C.  SYSTEMS  ENGINEERING  AND  THE  DOD  ACQUISITION  PROCESS 

Systems  engineering  is  a  critical  aspect  of  the  DoD  acquisition  process. 
Department  of  Defense  Instruction  (DoDI)  5000.02  mandates  that  systems  engineering  be 
integrated  into  the  DoD  acquisition  process,  stating: 

Systems  engineering  shall  be  embedded  in  program  planning  and  be 

designed  to  support  the  entire  acquisition  life  cycle.  [6] 
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Figure  2  shows  how  the  systems  engineering  technical  processes  described  above 
are  implemented  across  the  DoD  Defense  Acquisition  Management  System. 
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FIGURE  2.  SYSTEMS  ENGINEERING  PROCESS  AND  DEFENSE  ACQUISITION 

FRAMEWORK  [AFTER  6] 

D.  TEST  AND  EVALUATION  IN  THE  SYSTEMS  ENGINEERING  PROCESS 

The  terms  Test  and  Evaluation  are  often  used  interchangeably.  However,  while 
related,  the  terms  test  and  evaluation  have  two  distinct  definitions.  Testing  is  the  process 
of  obtaining  and  providing  data.  Evaluation  is  the  process  of  analyzing  the  data  that 
testing  produces. 

T&E  is  incorporated  throughout  the  systems  engineering  process.  T&E  is  the 
mechanism  for  accomplishing  verification  in  the  systems  engineering  process  and 
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characterizing  technical  risk  of  achieving  a  proper  final  design  solution.  While  some 
steps  of  the  systems  engineering  technical  processes  such  as  verification  and  validation 
are  obviously  T&E  related,  T&E  is  a  critical  and  continuous  activity  throughout  the 
systems  engineering  process  to  ensure  that  cost,  schedule,  and  performance  requirements 
are  satisfied  with  acceptable  levels  of  risk. 

At  each  stage  of  the  systems  engineering  process,  T&E  confirms  that  people, 
product,  and  process  solutions  meet  or  exceed  the  user’s  requirements.  T&E  provides  the 
feedback  loop  to  the  systems  engineering  process.  During  early  stages  of  the  systems 
engineering  process  T&E  activities  may  take  the  form  of  analysis,  modeling,  simulation, 
and  proof  of  concept  tests  for  system,  subsystem,  and  component  levels.  Later  stages 
will  focus  more  on  examination,  demonstration,  and  testing  to  verify  the  function  of  the 
design  and  validate  the  produced  systems  meets  user  requirements. 

T&E  is  involved  from  the  beginning  of  the  systems  engineering  process  during 
requirements  analysis  and  logical  analysis.  For  DoD  systems,  the  requirements  flow 
from  the  user  in  the  form  of  the  Initial  Capabilities  Document  (ICD),  followed  by  the 
Capabilities  Development  Document  (CDD),  and  finally  the  Capabilities  Production 
Document  (CPD),  with  the  requirements  becoming  more  mature  and  detailed  with  each 
successive  iteration.  System  perfonnance  requirements  are  described  in  terms  of 
attributes,  which  are  characteristics  that  describe  an  aspect  of  a  system  capability.  T&E 
input  is  important  during  these  stages  to  develop  good  requirements.  Specifically,  T&E 
input  is  critical  to  ensure  that  all  requirements  are  testable.  Test  techniques  and 
laboratory/range  capabilities  must  exist  or  must  be  capable  of  being  developed  to  test  to 
the  requirement.  While  a  requirement  may  sound  good  on  paper,  it  is  of  no  value  if  it 
cannot  be  verified  through  practical  testing.  T&E  may  also  be  used  to  refine 
requirements  and  concepts  of  operations  by  modeling,  simulation,  and  prototype  testing. 
T&E  involvement  is  also  important  to  give  the  T&E  community  insight  to  enable 
development  and  execution  of  a  successful  T&E  strategy. 

During  the  design  solution  phase  of  the  systems  engineering  process,  T&E  may 

take  the  form  of  Modeling  and  Simulation  (M&S).  During  early  phases  of  the  acquisition 

process  (prior  to  Milestone  B),  T&E  in  the  form  of  M&S  provides  an  evaluation  of 
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system  concepts  and  technology  alternatives  using  early  performance  parameter 
objectives  and  thresholds.  By  modeling  different  proposed  designs,  perfonnance  can  be 
predicted  and  design  tradeoffs  can  be  accomplished,  long  before  the  system  is  even  built. 

During  implementation,  integration,  and  verification,  T&E  is  conducted 
continually  to  detennine  if  the  produced  items  perform  as  they  were  designed.  During 
implementation,  component  level  testing  verifies  the  performance  of  each  component. 
This  is  followed  by  subassembly  testing  during  integration,  as  components  are  brought 
together  to  form  subsystems.  Verification  entails  full-system  Developmental  Test  and 
Evaluation  (DT&E)  to  verify  system  performance  against  the  system  design 
specifications.  DT&E  is  used  to  assist  engineering  design,  system  development,  risk 
identification,  and  evaluate  the  contractor’s  ability  to  attain  desired  technical  performance 
in  system  specifications.  DT&E  is  normally  performed  by  engineers,  technicians,  and 
contractors  in  a  controlled  environment  to  verify  that  design  requirements  are  met.  This 
testing  is  often  accomplished  on  a  developmental  test  article.  The  results  of 
developmental  tests  feed  back  into  the  design  solution  process.  Normally,  DT&E  is 
initially  conducted  by  the  developing  contractor  during  system  design  and  transitioned  to 
combined  contractor  and  government  DT&E  as  the  system  design  matures. 

The  validation  process  determines  whether  the  system  satisfactorily  meets  the 
user’s  requirement  through  Operational  Test  and  Evaluation  (OT&E).  OT&E  may  be 
accomplished  early-on  through  combined  DT&E  and  OT&E  or  Integrated  Test  (IT)  in 
order  to  give  the  operational  tester  an  early  look  at  the  system  and  maximize  efficiencies 
gathering  data  that  both  DT&E  and  OT&E  testers  have  an  interest  in.  OT&E  is  normally 
performed  by  the  end-user  of  the  system  on  a  production  representative  test  article  in  a 
realistic  operational  environment  where  the  system  must  interact  with  the  enviromnent, 
personnel,  threat,  interoperable  systems,  doctrine  and  tactics  to  validate  that  the  user’s 
requirements  are  met.  The  results  of  OT&E  feed  back  into  the  requirements  development 
and  design  solution  processes  if  and  when  deficiencies  are  discovered. 

Finally,  additional  DT  and  Follow-on  Operational  Test  and  Evaluation  (FOT&E) 
is  conducted  during  transition  to  full  operation  to  test  system  components  that  could  not 

be  fully  tested  during  verification  and  validation  and  to  test  new  upgrades  to  the  system. 
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DT  and  FOT&E  is  also  used  to  test  future  increments,  modifications,  and  upgrades  and 
help  refine  doctrine,  tactics,  techniques,  and  training  programs. 


Figure  3  graphically  illustrates  the  role  of  T&E  throughout  the  systems 
engineering  process. 


FIGURE  3.  ROLE  OF  T&E  IN  THE  SYSTEMS  ENGINEERING  PROCESS 
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III.  T&E  GUIDANCE  REVIEW 


A.  INTRODUCTION 

This  chapter  will  answer  the  research  question, 

What  guidance  is  currently  provided  by  Department  of  Defense  (DoD), 

Department  of  Navy  (DoN),  and  Naval  Air  Systems  Command 

(NAVAIR)  regarding  integrating  T&E  into  the  systems  acquisition 

process? 

T&E  direction  for  Naval  Aviation  acquisition  programs  flows  from  federal  law  to 
DoD,  to  DoN,  to  NAVAIR  in  the  form  of  statutory,  regulatory,  and  discretionary 
guidance.  At  each  level,  the  statutory  guidance  tells  what  must  be  done  by  law,  the 
regulatory  guidance  tells  what  must  be  done  by  policy,  and  the  discretionary  guidance 
tells  how  it  should  be  done  as  learned  from  experience. 

This  chapter  will  review  T&E  guidance  that  is  currently  guiding  Naval  Aviation 
acquisition  down  through  the  chain  of  federal  law,  DoD,  DoN,  and  NAVAIR.  This 
chapter  will  then  review  T&E  guidance  from  other  organizations  and  services  within 
DoD. 

B.  FEDERAL  LAW  T&E  GUIDANCE 

Title  10  of  U.S.  Code  dictates  statutory  requirements  for  the  conduct  and 
oversight  of  operational  testing. 

Specifically,  Section  139  establishes  the  office  of  Director  Operational  Test  and 
Evaluation  (DOT&E)  to  serve  as  the  principal  adviser  to  the  Secretary  of  Defense  and  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  on  Operational 
Test  and  Evaluation  (OT&E)  in  DoD  and  the  principal  OT&E  official  within  the  senior 
management  of  DoD  [7].  Title  10  outlines  responsibilities  of  DOT&E,  including 
providing  oversight  to  operational  test  planning  and  execution. 
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Section  2399  of  Title  10  defines  the  timing,  conduct,  and  reporting  requirements 
of  Operational  Test  (OT)  in  support  of  DoD  acquisition  programs  [7].  Title  10  requires 
that  completion  of  Initial  Operational  Test  and  Evaluation  (IOT&E)  of  a  major  defense 
acquisition  program  is  required  prior  to  proceeding  beyond  Low-Rate  Initial  Production 
(LRIP).  The  DOT&E  must  approve  the  adequacy  of  OT&E  and  submit  a  report  at  the 
conclusion  of  OT. 

Finally,  Section  2366  of  Title  10  requires  major  systems  and  munitions  programs 
to  undergo  survivability  testing  and  lethality  testing,  otherwise  known  as  Live-Fire  Test 
and  Evaluation  (LFT&E)  prior  to  full-rate  production  [7]. 

C.  DOD  T&E  GUIDANCE 

1.  Defense  Acquisition  System,  DoD  Directive  5000.1 

DoD  Directive  5000.1  provides  management  principles  and  mandatory  policies 
and  procedures  for  managing  all  acquisition  programs  [8].  Regarding  T&E,  DoD 
Directive  5000.1  provides  two  mandatory  policies. 

First,  it  requires  that  each  military  branch  establish  an  independent  operational 
test  agency  to  plan  and  conduct  operational  tests,  report  results,  and  provide  evaluations 
of  effectiveness  and  suitability. 

Second,  it  requires  that  T&E  be  integrated  throughout  the  defense  acquisition 
process.  It  gives  direction  to  the  purpose  of  T&E  in  the  defense  acquisition  process, 
stating: 

Test  and  evaluation  shall  be  structured  to  provide  essential  information  to 
decision-makers,  assess  attainment  of  technical  perfonnance  parameters, 
and  detennine  whether  systems  are  operationally  effective,  suitable, 
survivable,  and  safe  for  intended  use.  The  conduct  of  test  and  evaluation, 
integrated  with  modeling  and  simulation,  shall  facilitate  learning,  assess 
technology  maturity  and  interoperability,  facilitate  integration  into  fielded 
forces,  and  confirm  perfonnance  against  documented  capability  needs  and 
adversary  capabilities  as  described  in  the  system  threat  assessment.  [8] 
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2.  Operation  of  the  Defense  Acquisition  System,  DoDI  5000.02 


DoDI  5000.02  provides  regulatory  requirements  for  the  operation  of  the  defense 
acquisition  system  [6].  In  addition  to  the  statutory  OT  and  LFT&E  requirements  of  Title 
10,  DoDI  5000.02  provides  regulatory  T&E  planning  and  reporting  requirements 
including  development  of  the  Component  LFT&E  Report,  the  Operational  Test  Agency 
(OTA)  Report  of  OT&E  Results,  the  Test  and  Evaluation  Master  Plan  (TEMP),  and  the 
Test  and  Evaluation  Strategy  (TES). 

DoDI  5000.02  emphasizes  the  importance  of  early  involvement  of  the  T&E 
community  in  the  systems  acquisition  process.  Direction  is  given  to  bring  T&E  expertise 
“to  bear  at  the  beginning  of  the  system  life  cycle  ...  so  that  appropriate  and  timely 
corrective  actions  can  be  developed  prior  to  fielding  the  system  [6].” 

DoDI  5000.02  sets  the  following  T&E  related  regulatory  requirements: 

•  Integration  of  all  T&E  activities  into  an  efficient  continuum 

•  Responsibility  of  the  Program  Manager  (PM)  to  design  DT&E  objectives 
appropriate  to  each  phase  and  milestone  of  an  acquisition  program 

•  TES  development 

•  TEMP  development 

•  T&E  planning  requirements 

•  DT&E  requirements 

•  Readiness  for  OT&E 

•  OT&E  requirements 

•  LFT&E  requirements 

•  Use  of  M&S  throughout  the  acquisition  lifecycle 

•  Foreign  comparative  testing  requirements 

•  T&E  of  evolutionary  acquisition  programs 
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3. 


Defense  Acquisition  Guidebook 


The  purpose  of  the  DAG  is  to  complement  the  regulatory  guidance  of  DoD 
Directive  5000.1  and  DoDI  5000.02  by  providing  the  acquisition  workforce  with 
discretionary  best  practices  [1].  While  discretionary  in  nature,  the  DAG  provides  non¬ 
mandatory  DoD  staff  expectations  for  satisfying  the  mandatory  requirements  of  DoD 
Directive  5000.1  and  DoDI  5000.02. 

Chapter  9  of  the  DAG  is  focused  on  Integrated  Test  and  Evaluation  (IT&E), 
giving  discretionary  guidance  on  how  to  fulfill  the  mandatory  requirements  of  the  DoD 
5000-series  policy. 

The  DAG: 

•  Provides  and  introduction  of  general  topics  associated  with  T&E 

•  Presents  an  overview  of  T&E  support  and  oversight  provided  by  the 
Office  of  the  Secretary  of  Defense  (OSD) 

•  Describes  the  relationship  of  Joint  Capabilities  Integration  and 
Development  System  (JCIDS)  documents  to  T&E 

•  Explains  the  philosophy  behind  IT 

•  Provides  guidelines  for  conduct  of  DT&E 

•  Describes  the  purpose  and  value  of  a  Test  and  Evaluation  Working 
Integrated  Product  Team  (T&E  WIPT) 

•  Provides  best  practices  for  the  use  of  M&S  in  DT&E 

•  Provides  guidelines  for  conduct  of  OT&E 

•  Gives  guidance  for  the  development  of  the  TES  and  TEMP 

•  Presents  guidance  for  the  preparation  of  mandatory  T&E  reports 

•  Discusses  T&E  best  practices 
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D.  DON  T&E  GUIDANCE 


1.  Implementation  and  Operation  of  the  Defense  Acquisition  System  and 
the  Joint  Capabilities  Integration  and  Development  System, 
SECNAVINST  5000.2D 

Secretary  of  the  Navy  Instruction  (SECNAVINST)  5000. 2D  provides  mandatory 
procedures  for  DoN  implementation  of  DoD  5000-series  acquisition  and  JCIDS  policy 
[9].  Chapter  5  of  SECNAVINST  is  focused  on  T&E.  This  chapter  delineates  the 
mandatory  roles,  responsibilities,  procedures,  and  requirements  for  DoN  acquisition 
programs. 

SECNAVINST  5000.2D  addresses: 

•  DoN  Responsibilities  for  T&E 

•  T&E  Strategy  Requirements 

•  T&E  Planning  Requirements 

•  DT&E  Requirements 

•  Certification  of  Readiness  for  Operational  Testing 

•  OT&E  Requirements 

•  The  Annual  OSD  T&E  Oversight  List 

•  LFT&E  Requirements 

•  Comparative  Testing  Requirements 

•  T&E  Reporting  Requirements 

2.  DoN  Acquisition  and  Capabilities  Guidebook 

The  DoN  Acquisition  and  Capabilities  Guidebook  is  a  companion  document  to 
SECNAVINST  5000. 2D  [10].  Following  the  same  chap  ter/ enclosure/paragraph 
structure,  the  Guidebook  incorporates  all  the  infonnation  given  in  the  Instruction,  adding 
discretionary  guidance  to  the  mandatory  guidance  of  the  instruction. 
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E. 


NAVAIR  T&E  GUIDANCE 


Acquisition  Test  and  Evaluation,  NAVAIR  Instruction  (NAVAIRINST)  3960.2D 
states  policy,  assigns  responsibilities  and  provides  procedures  for  acquisition-related 
T&E  activities  of  programs  and  systems  managed  by  NAVAIR  [11].  Just  as 
SECNAVINST  5000.2D  provides  mandatory  procedures  for  DoN  implementation  of 
DoD  5000-series  acquisition  policy,  the  NAVAIRINST  3960.2D  provides  mandatory 
procedures  for  NAVAIR  implementation  of  both  the  DoD  and  SECNAV  level  policy 
relating  to  T&E. 

NAVAIRINST  3960.2D  provides  guidance  in  the  following  areas: 

•  Outlines  membership,  frequency,  and  conduct  of  T&E  WIPTs 

•  TEMP  preparation  procedures 

•  Conduct  of  the  T&E  Executive  Strategy  Review  (T&E  ESR) 

•  Policies  and  procedures  for  planning  and  conduct  of  the  Operational  Test 
Readiness  Review  (OTRR)  and  pre-OTRR 

•  Process  for  Fleet  Research,  Development,  Test  and  Evaluation  (RDT&E) 
Support  requests 

F.  DEFENSE  ACQUISITION  UNIVERSITY  (DAU)  T&E  GUIDANCE 

The  T&E  Management  Guide  published  by  DAU  is  a  non-Service  specific 
technical  management  educational  guide  [12].  The  Guide  is  intended  primarily  for  use  in 
courses  taught  by  DAU,  but  is  also  useful  as  a  desk  reference.  The  T&E  Management 
Guide  provides  infonnation  on  the  following  areas: 

•  T&E  contributions  leading  to  each  milestone 

•  A  summary  of  T&E  activities  during  each  phase  of  the  acquisition  process 

•  A  summary  of  T&E  documents  developed  during  each  phase  of  the 

acquisition  process 

•  A  description  of  T&E  within  the  systems  engineering  process 
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•  T&E  policy  structure  and  oversight  mechanisms 

•  Program-office  responsibilities  for  T&E 

•  M&S  to  support  T&E 

•  T&E  resources 

•  Software  T&E 

•  LFT&E 

•  Logistics  T&E 

•  Multi-service  T&E 

•  International  T&E  programs 

•  Commercial  and  Non-Developmental  Items  (NDI)  T&E 

G.  T&E  GUIDANCE  FROM  OTHER  SERVICES 

1.  Air  Force  T&E  Guidance 

a.  T&E  Process,  Air  Force  Policy  Directive  AFPD  99-1 

Air  Force  Policy  Directive  (AFPD)  99-1  establishes  Air  Force  policies  for 
the  T&E  process  and  infrastructure  [13].  Specifically,  this  directive: 

•  Establishes  policy  requirements  for  DT&E,  Qualification  Test  and 
Evaluation  (QT&E),  Contractor  Testing  (CT),  Initial  Operational  Test 
and  Evaluation  (IOT&E),  andFOT&E 

•  Requires  operation,  maintenance  and  improvements  to  T&E  facilities 

•  Provides  a  requirement  for  the  Weapon  System  Evaluation  Program 
(WSEP) 

•  Establishes  T&E  related  responsibilities  and  authorities 
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b.  Capabilities-Based  Test  and  Evaluation,  AFI 99-1 03 


Air  Force  Instruction  (AFI)  99-103  implements  AFPD  99-1  by  describing 
planning,  conduct,  and  reporting  of  cost  effective  T&E  programs  as  an  efficient 
continuum  of  integrated  testing  known  as  seamless  verification  [14].  Specifically,  this 
instruction  provides  Air  Force  policy  regarding: 

•  Air  Force  vision  and  implementation  concepts 

•  Types  of  T&E 

•  T&E  responsibilities 

•  T&E  activities  supporting  each  milestone 

•  T&E  oversight  and  reporting 

c.  Air  Force  T&E  Guidebook 

The  Air  Force  T&E  Guidebook  contains  information,  guidance,  best 
practices,  and  lessons-learned  about  T&E  and  related  subjects  that  were  not  published  in 
AFI  99-103  [15].  Discretionary  in  nature,  it  is  intended  to  supplement  and  expand  on  the 
policies  and  guidance  of  AFPD  99-1  and  AFI  99-103.  Specifically,  the  Guidebook 
provides  guidance  in  the  following  areas: 

•  Hierarchy  of  T&E  policy  and  guidance 

•  Relationships  with  OSD 

•  T&E  support  to  the  requirements  process 

•  T&E  support  to  the  acquisition  process 

•  The  IT  process 

•  Integrated  Test  Team  (ITT)  tools  and  techniques 

•  ITT  products 

•  DT&E 
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•  OT&E 


•  Space  Systems  T&E 

•  T&E  resources 

•  Deficiency  reporting 

2.  Army  T&E  Guidance 

a.  Test  and  Evaluation  Policy,  AR  73-1 

Army  Regulation  (AR)  73-1  implements  the  policies  and  procedures 
contained  in  the  DOD-5000  series  and  DAG  and  specifically  prescribes  implementing 
policies  for  the  Army’s  testing  and  evaluation  program  [16].  AR  73-1  provides  Army- 
specific  policy  on  the  following  areas: 

•  T&E  roles  and  responsibilities 

•  T&E  in  support  of  systems  acquisition  and  development 

•  DT,  OT,  and  Evaluation 

•  T&E  WIPTs 

•  Conduct  of  the  Test  Schedule  and  Review  Committee 

•  T&E  review  and  reporting  requirements 

•  T&E  budget  and  financial  considerations 

b.  Test  and  Evaluation  in  Support  of  Systems  Acquisition,  DA  PAM 
73-1 

Department  of  the  Anny  Pamphlet  (DA  PAM)  73-1  implements  the 
policies  contained  in  AR  73-1.  Specifically  it  [17]: 

•  Provides  an  overview  of  the  T&E  process  in  support  of  Army 
acquisition  systems 

•  Describes  the  T&E  WIPT 
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•  Provides  detailed  guidance  and  procedures  for  the  preparation, 
staffing,  and  approval  of  the  TEMP 

•  Provides  an  overview  of  the  Army  Critical  Operational  Issues  and 
Criteria  development  and  approval  processes 

•  Provides  an  overview  of  the  Army  System  Evaluation  and  System 
Assessment  proves 

•  Provides  an  overview  of  Army  developmental  and  operational  testing 
processes 
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IV.  ANALYSIS  OF  THE  CURRENT  GUIDANCE 


A.  INTRODUCTION 

This  chapter  and  the  following  chapter  will  answer  the  remaining  research 
questions: 

1.  What  areas  of  weakness  exist  regarding  integrating  T&E  into  the  systems 
acquisition  process? 

2.  What  improvements  can  be  made  to  policies,  procedures,  and  guidance  to 
better  integrate  T&E  into  the  systems  acquisition  process? 

As  seen  from  the  previous  chapter,  Naval  Aviation  weapons  systems  acquisition 
guidance  and  policy  flows  downward  from  Federal  law,  to  DoD,  to  DoN,  to  NAVAIR. 
This  guidance  and  policy  specifies  what  actions  need  to  take  place  during  the  acquisition 
process,  when  they  need  to  take  place,  and  how  they  should  be  accomplished. 

This  chapter  describes  how  T&E’s  integration  into  the  systems  acquisition 
process  is  guided  by  this  policy.  It  also  presents  results  of  the  author’s  analysis  of 
weaknesses  in  guidance  and  policy  and  recommends  steps  for  improvement. 

B.  REVIEW  OF  POLICY  AND  GUIDANCE  GOVERNING  T&E’S 

INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS 

Tables  1-9  illustrate  T&E  involvement  at  significant  events  along  the  acquisition 
process.  These  tables  show  the  phases  of  the  defense  acquisition  framework  along  the 
vertical  axis,  with  major  events  and  tasks  involving  T&E  input  noted  within  each  phase. 
For  each  event,  the  input  that  T&E  provides  along  with  the  governing  guidance  from 
DoD,  DoN,  NAVAIR,  and  other  sources  is  noted.  Each  event  or  task  is  assessed  as 
Good,  Marginal,  or  Poor.  Good  means  that  the  guidance  sufficiently  addresses  T&E’s 
role  in  the  event  or  task.  Marginal  means  that  the  guidance  addresses  T&E’s  role,  but 
that  the  guidance  is  incomplete  and  could  still  be  improved  upon.  Poor  means  that  the 
guidance  either  does  not  address  T&E’s  role  at  all,  or  is  severely  insufficient.  The 
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assessments  of  Good,  Marginal,  and  Poor  are  color-coded  green,  yellow,  and  red, 
respectively.  Acquisition  phases  in  the  first  column  are  color  coded  to  match  the  code 
used  for  program  phases  in  Figure  2. 

The  author’s  assessments  of  guidance  quality  are  based  on  the  rationale  provided 
in  the  Notes/Comments  column.  The  assessment  is  based  on  the  author’s  opinion  from 
review  of  the  guidance  along  with  views  expressed  from  the  interview  subjects.  The 
interviews  are  discussed  in  detail  in  the  next  chapter. 
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Acq  Phase 


Event  or  Task 


Initial  Capabilities 
Document  (ICD) 
Development 


Analysis  of 
Alternatives  (AoA) 


Technology 
Development 
Strategy  (IDS) 


Request  for 
Proposals  (RFP) 


T&E  Input 


Provide  Subject  Matter  Experts  (SMEs) 
for  requirements  testability  and  review  of 
the  ICD  for  reasonable  requirements 


Review  alternatives  to  develop 
applicable  T&E  strategiy  to  each 
alternative. 


Provide  T&E  input  to  preliminary 
acquisition  strategy,  including  overall 
cost,  schedule,  and 

performance  goals  for  the  total  research 
and  development  program 


Notional  T&E  requirements  and  strategy 


GOVERNING  GUIDANCE 


Assessment  of 
Guidance 


DoD 


Joint  Capabilities 
Integration  and 
Development 
System,  CJCSI 
3170.01  G 
[Reference  18] 


DoDI  5000.02 
Enel  7,  Para.  5 


DoDI  5000.02, 
Enel  2,  Para. 
5.C.7 


Federal 
Acquisition 
Regulation  (FAR) 
[Reference  20]; 
DoDI  5000.02 


DoN 


SECNAVINST 
5000. 2D 


SECNAVINST 
5000. 2D,  Para. 
6.4 


SECNAVINST 
5000. 2D,  para. 3. 4 


SECNAVINST 
5000. 2D 


NAVAIR 


NAVAIR 

Acquisition  Guide 
[Reference  19], 
Para.  5.3B 


NAVAIR 
Acquisition 
Guide,  Para  5.4 


NAVAIRINST 
4200. 39 B 
[Reference  21  ] 


OTHER 


Notes  Comments 


|The  CJCSI  does  not  define  responsibilities 
[below  the  Services.  The  SECNAVINST 
[requires  the  draft  ICD  to  be  forwarded  to  the 
IsYSCOM  for  information  only.  N091  is  the 
[Navy  organization  responsible  for  review  of 
IlCD  for  testability  and  resolving  test  issues. 


iNo  mention  of  T&E's  role  in  the  development 
I  of  the  AOA  is  given  in  any  of  the 
Iguidance/policy  documents. 


[The  SECNAVINST  states  that  the  PM  is 
[responsible  for  developing  the  TDS,  however  it 
[gives  no  guidance  regarding  who  it  should  be 
Icoordinated  with. 


[No  mention  of  T&E  in  the  NAVAIR  instruction. 
[The  SECNAVINST  states  that  N091  is  in  an 
[advisory  capacity. 


TABLE  1 .  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  1  OF  9) 
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Acq  Phase 

Event  or  Task 

T&E  Input 

GOVERNING  GUIDANCE 

Assessment  of 
Guidance 

Notes  Comments 

DoD 

DoN 

NAVAIR 

OTHER 

Source  Selection 

T&E  Strategies  for  competing 
Alternatives  -  cost,  schedule  and 
performance 

FAR.  DoDI 
5000.02 

SECNAVINST 
5000. 2D 

NAVAIRINST 
4200. 39 B 

No  mention  of  T&E  in  the  NAVAIR  instruction. 
The  SECNAVINST  states  that  N091  is  in  an 
advisory  capacity. 

I 

< 

</> 

c 

o 

w 

’3 

cr 

0 

Initial  Technical 
Review  (ITR) 

Notional  test  requirements,  cost 
estimate;  initial  schedule 

Naval  Systems 
Engineering 
Resource  Center 
(NSERC)  SETR 
Checklist, 
[Reference  22] 

NAVAIRINST 
4355.19D 
[Reference  23] 

Assistant  Program  Manager  for  Test  and 
Evaluation  (APMT&E)  identified  on  technical 
review  board  to  ensure  that  all  T&E 
requirements  are  addressed.  However,  no 
guidance  exists  for  specific  T&E  participation. 

_c 

"o 

(/) 

S 

S 

£ 

0) 

0? 

°? 

Qj 

CL 

Alternative 
Systems  Review 
(ASR) 

Notional  test  requirements,  cost 
estimate;  initial  schedule,  Draft  T&E 
Strategy. 

NSERC  SETR 
Checklist 

NAVAIRINST 

4355.19D 

Marginal 

APMT&E  identified  on  technical  review  board 
to  ensure  that  all  T&E  requirements  are  met. 
However,  no  guidance  exists  for  specific  T&E 
participation.  T&E  requirements  for  alternative 
systems  not  well  addressed  in  ASR  checklist 

Test  and 
Evaluation 
Strategy  (TES) 

Overall  test  approach  for  integrating 
DT&E,  OT&E,  and  LFT&E  and 
addresses  resource  planning 

DoDINST 
5000.  02,  Enel  6, 
Para.  2. a; 
DAG,  Para.  9.6.1 

SECNAVINST 
5000. 2D,  Para. 
5.3 

DAG  provides  guidance  for  contents  and 
format  of  TES. 

TABLE  2.  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  2  OF  9) 
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Acq  Phase 

Event  or  Task 

T&E  Input 

GOVERNING  GUIDANCE 

Assessment  of 
Guidance 

Notes  Comments 

DoD 

DoN 

NAVAIR 

OTHER 

© 

3  </> 

<s| 

s< 

13 

S 

Systems 
Engineering  Plan 
(SEP) 

T&E  role  in  the  systems  engineering 
process 

DoDINST 
5000.02,  Enel. 
12,  Para.  2; 
DAG,  Para.  4.5.1; 
USD(AT&L)  SEP 
Preparation  Guide 
(Reference  24] 

ASN(RD&A) 
Memorandum  of 
IB  November 
2007  [Reference 
25] 

SEP  Preparation  Guide  emphasizes 
documenting  relationships  between  systems 
engineering  and  test  and  evaluation 

MSA 


Capabilities 
Development 
Document  (CDD) 

Provide  Subject  Matter  Experts  (SMEs) 
for  requirements  testability  and  review  of 
the  CDD  for  reasonable  requirements 

CJCSI  3170.01  G 

SECNAVINST 
5000. 2D 

The  CJCSI  does  not  define  responsibilities 
below  the  Services.  The  SECNAVINST 
requires  the  draft  CDD  to  be  forwarded  to  the 
SYSCOM  for  information  only.  N091  is  the 
Navy  organization  responsible  for  review  of 

CDD  for  testability  and  resolving  test  issues. 

System 
Requirements 
Review  (SRR)  - 1 

M&S  role  in  testing  identified, 
requirement  verification  methodology, 
preliminary  test  requirements,  updated 
cost  estimate,  preliminary  schedule, 
Draft  TEMP. 

NSERC  SETR 
Checklist 

NAVAIRINST 

4355.19D 

Marginal 

DT  and  OT  personnel  identified  as  technical 
review  participants.  However,  no  guidance 
exists  for  specific  T&E  participation. 

SRR  -  II 

M&S  role  in  testing  identified,  Draft 
Verification  and  Validation  (V&V) 
methodology  defined  for  each 
Specification  requirement,  Draft  TEMP 
updated,  updated  requirement 
verfication  methodology 

NSERC  SETR 
Checklist 

NAVAIRINST 

4355.19D 

Marginal 

DT  and  OT  personnel  identified  as  technical 
review  participants.  However,  no  guidance 
exists  for  specific  T&E  participation. 

TABLE  3.  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  3  OF  9) 
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Acq  Pluise 

Event  or  Task 

T&E  Input 

GOVERNING  GUIDANCE 

Assessment  of 
Guidance 

Notes  Comments 

DoD 

DoN 

NAVAIR 

OTHER 

Technology  Development 

j  Pre-Systems  Acquisition 

Formation  ofT&E 
WIPT 

The  T&E  WIPT  serves  as  the  forum  for 
planning  and  cooridinating  the  T&E 
activities  of  the  program,  fostering 
cooridination  and  communication 
between  DT,  OT,  and  other  T&E 
stakeholders 

DAG,  Chap.  9 

SECNAVINST 
5000. 2D,  Para. 
5.4.3 

NAVAIRINST 
3960. 2D,  Enel.  1 

Good 

NAVAIRINST  3960.2D  Enel.  1  provides 
guidance  outlining  the  membership,  frequency 
and  conduct  ofT&E  WIPTs. 

System  Functional 
Review  (SFR) 

Functional  requirements  assigned  to 
M&S,  Draft  software  test  plan,  if  not  a 
software  intensive  program  (and  no  SSR 
is  planned) 

NSERC  SETR 
Checklist 

NAVAIRINST 
4355. 19D 

Marginal 

DT  and  OT  personnel  identified  as  technical 
review  participants.  However,  no  guidance 
exists  for  specific  T&E  participation. 

TEMP 

Planned  DT&E,  OT&E,  and  LFT&E, 
including  measures  to  evaluate  the 
performance  of  the  system  during  these 
test  periods;  an  integrated  test 
schedule;  and  the  resource 
requirements  to  accomplish  the  planned 
testing. 

DoDINST 

5000  . 02,  Enel  6, 
Para.  2.b, 
DAG,  Para  9.6.2 

SECNAVINST 
5000. 2D,  Para. 
5.3 

NAVAIRINST 
3960. 2D,  Enel.  2 

Good 

DAG  provides  guidance  for  contents  of  TEMP. 
NAVAIRINST  3960.2D,  Enel.  2  provides 
NAVAIR  specific  policy. 

Acquisition 

Strategy 

Provide  T&E  input  to  time-phased 
workload  assessment  identifying  the 
manpower  and  functional  competency 
requirements  for  successful  program 
execution  and  the  associated  staffing 
plan,  including  the  roles  of  government 
and  non-government  personnel 

DoDI  5000.02, 
Enel  2,  Para 
6.d.7; 

DAG,  Para.  2.3 

SECNAVINST 
5000. 2D,  Para.3.4 

NAVAIR 
Acquisition 
Guide,  Para  8. A 

Good 

The  DAG  states  "The  program  manager 
should  engage  the  Test  and  Evaluation 
Working-Level  Integrated  Product  Team  in  the 
development  of  the  acquisition  strategy,  and 
harmonize  the  acquisition  strategy  and  the 

Test  and  Evaluation  Strategy." 

TABLE  4.  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  4  OF  9) 
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Acq  Phase 

Event  or  Task 

T&E  Input 

GOVERNING  GUIDANCE 

Assessment  of 
Guidance 

Notes  Comments 

DoD 

DoN 

NAVAIR 

OTHER 

Technology 

Development 

^  Pre-Systems 

Acquisition 

Capabilities 
Production 
Document  (CPD) 

Provide  Subject  Matter  Experts  (SMEs) 
for  requirements  testability  and  review  of 
the  CPD  for  reasonable  requirements 

CJCSI  31 70.01  G 

SECNAVINST 
5000. 2D 

The  CJCSI  does  not  define  responsibilities 
below  the  Services.  The  SECNAVINST 
requires  the  draft  CPD  to  be  forwarded  to  the 
SYSCOM  for  information  only.  N091  is  the 
Navy  organization  responsible  for  review  of 

CPD  for  testability  and  resolving  test  issues. 

MS  B 


Preliminary  Design 
Review  (PDR) 

Test  verfication  matrix  covering  sub¬ 
system  allocations,  traceability  from 
design  documentation  to  sub-system 
test  requirements,  identfication  of 
engineering  data  requirements  needed 
from  testing 

NSERC  SETR 
Checklist 

NAVAIRINST 

4355.19D 

Marginal 

APMT&E  identified  on  technical  review  board 
to  ensure  that  all  T&E  requirements  are 
addressed.  However,  no  guidance  exists  for 
specific  T&E  participation. 

Critical  Design 
Review  (CDR) 

Test  verfication  matrix,  engineering  data 
requirements  from  testing,  test  plans, 
M&S  role  in  testing  defined,  finalized 
V&V  of  Systems  Integraton  Lab  (SIL) 
plan 

NSERC  SETR 
Checklist 

NAVAIRINST 

4355.19D 

Marginal 

APMT&E  identified  on  technical  review  board 
to  ensure  that  all  T&E  requirements  are 
addressed.  However,  no  guidance  exists  for 
specific  T&E  participation. 

Operational 
Assessment  (OA) 

OATest  Plans  and  Reports 

Operational  Test 
Director's  Manual, 
COMOPTEV- 
FORINST  3980.1, 
[Reference  26] 

Establishes  policy  and  guidance  on  all 
aspects  of  OT&E,  including  planning,  conduct 
and  reporting. 

TABLE  5.  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  5  OF  9) 
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Acq  Phase 


Event  or  Task 


Risk  Management 


DTTest  Planning 


Test  Readiness 
Review  (TRR) 


Flight  Readiness 
Review  (FRR) 


T&E  Input 


Risk  elements  affecting  successful 
accomplishment  ofT&E  program  and 
risk  elements  identfied  by  T&E  results 


DT  Test  Plans 


Requirements  verification  matrix,  test 
plan,  test  and  validation  procedures, 
plan  for  test  reporting,  functional 
anomaly  reporting  system 


Finalized  engineering  data  requirements 
from  flight  test,  defined  data  reduction 
and  analysisroles  and  responsibilities, 
test  plans,  flight  test  requirements 
supporting  M&S  validation 


GOVERNING  GUIDANCE 


DoD 


Risk  Management 
Guide  for  DoD 
Acquisition  Sixth 
Edition 

[Reference  27] 


DoN 


NSERC  SETR 
Checklist 


NSERC  SETR 
Checklist 


NAVAIR 


NAVAIR  Risk 
Assessment 
Handbook 
[Reference  28] 


Project  Test  Plan 
Policy  and  Guide 
NAVAIRINST 
3960. 4  B 
[Reference  29] 


NAVAIRINST 
4355. 19D 


NAVAIRINST 
4355. 19D 


Assessment  of 
Guidance 


OTHER 


Notes  /  Comments 


iNo  guidance  given  on  T&E's  participation  in 
Ithe  risk  management  process. 


I  NAVAIRINST  3960.4B  establishes  NAVAIR 
I  policies ,  processes,  responsibilities,  and 
[requirements  for  preparation,  review,  and 
[execution  of  flight,  ground,  and  laboratory 
[tests  of  air  vehicles,  air  vehicle  weapons,  and 
[air  vehicle  installed  systems. 


I  Senior  AIR-5. 1  person  designated  to  serve  as 
[co-chair  of  review.  APMT&E  identified  on 
[technical  review  board  to  ensure  that  all  T&E 
[requirements  are  addressed.  A  TRR  must 
[have  an  appropriate  T&E  SME  representative 
Iforthe  system  under  test. 


I  AIR-5. 1  Test  Wing  Commander  designated  as 
[co-chair  or  review.  APMT&E  identified  on 
[technical  review  board  to  ensure  that  all  T&E 
[requirements  are  addressed. 


TABLE  6.  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  6  OF  9) 
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Acq  Phase 


MS  C 


-  o> 
5  - 
•s  >» 


Event  or  Task 

T&E  Input 

GOVERNING  GUIDANCE 

DoD 

DoN 

NAVAIR 

OTHER 

DT  Test  Reporting 

DTTest  Reports 

RDT&E  Report 
Writing  Handbook 
[Reference  30]; 
Director's  Note  1 , 
Test  Reporting 
[Reference  31  ] 

System 

Verification  Review 
(SVR) 

T&E  input  to  technical  review  checklist 
and  preparation 

NSERC  SETR 
Checklist 

NAVAIRINST 

4355.19D 

Production 
Readiness  Review 
(PRR) 

T&E  input  to  technical  review  checklist 
and  preparation 

NAVAIRINST 

4355.19D 

IOT&E  Planning 

OT  Test  Plans 

Operational  Test 
Director's  Manual, 
COMOPTEV- 
FORINST  3980.1 

Assessment  of 
Guidance 


Notes  /  Comments 


Report  Writing  Handbook  provides  formats 
and  guidance  for  reporting  test  results  from 
NAVAIR  test  events. 


APMT&E  identified  on  technical  review  board 
to  ensure  that  all  T&E  requirements  are 
addressed. 


APMT&E  identified  on  technical  review  board 
to  ensure  that  all  T&E  requirements  are 
addressed. 


Establishes  policy  and  guidance  on  all 
aspects  of  OT&E,  including  planning,  conduct 
and  reporting 


TABLE  7.  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  7  OF  9) 
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Acq  Phase 


Event  or  Task 


Operational  Test 
Readiness  Review 
(OTRR) 


IOT&E  Reporting 


Follow-on 
Developmental 
Test  and 
Evaluation 
Planning 


Follow-on 
Developmental 
Test  and 
Evaluation 
Reporting 


T&E  Input 


Completed  TEMP,  DT  test  reports  and 
results,  T&E  risk  identification 


OT  Test  Reports 


DT  Test  Plans 


DT  Test  Reports 


GOVERNING  GUIDANCE 


Assessment  of 
Guidance 


DoD 


DoN 


SECNAVINST 
5000. 2D,  Para. 
5.6 


NAVAIR 


NAVAIRINST 
3960. 2D,  Enel.  3 


NAVAIRINST 
3960. 4  B 


RDT&E  Report 
Writing  Handbook 
Director's  Note  1 , 
Test  Reporting 


OTHER 


Operational  Test 
Director's  Manual, 
COMOPTEV- 
FORINST  3980.1 


Notes  /  Comments 


I  NAVAIRINST  3960.2D  further  defines  the 
(requirements  of  SECNAVINST  5000. 3D, 
(which  establishes  the  minimum  criteria 
I  required  for  certification  of  readines  to 
(commence  operational  testing. 


lOT  Director's  Manual  establishes  policy  and 
(guidance  on  all  aspects  of  OT&E,  including 
(planning,  conduct  and  reporting 


I  NAVAIRINST  3960.4B  establishes  NAVAIR 
(policies,  processes,  responsibilities,  and 
(requirements  for  preparation,  review,  and 
(execution  of  flight,  ground,  and  laboratory 
(tests  of  air  vehicles,  air  vehicle  weapons,  and 
lair  vehicle  installed  systems. 


I  Report  Writing  Handbook  provides  formats 
(and  guidance  for  reporting  test  results  from 
|  NAVAIR  test  events. 


TABLE  8.  T&E  INTEGRATION  INTO  THE  SYSTEMS  ACQUISITION  PROCESS  (TABLE  8  OF  9) 
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Acq  Phase 

Event  or  Task 

T&E  Input 

GOVERNING  GUIDANCE 

Assessment  of 
Guidance 

Notes  Comments 

DoD 

DoN 

NAVAIR 

OTHER 

o 

Follow-on 
Operational  Test 
and  Evaluation 
(FOT&E)  Planning 

OTTest  Plans 

Operational  Test 
Director's  Manual, 
COMOPTEV- 
FORINST  3980.1 

Good 

OT  Director's  Manual  establishes  policy  and 
guidance  on  all  aspects  of  OT&E,  including 
planning,  conduct  and  reporting 

(75 

c 

CD 

E 

c 

</> 

© 

13 

5> 

© 

CO 

"55 

3 

CO 

FOT&E  Reporting 

OT  Test  Reports 

Operational  Test 
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c. 


ANALYSIS  RESULTS 


From  review  of  Tables  1-9,  it  is  evident  that  guidance  governing  certain  phases, 
events,  and  tasks  of  the  systems  acquisition  process  better  address  the  integration  of  T&E 
than  others.  As  an  overall  trend,  Tables  1-9  show  that  guidance  governing  T&E 
integration  improves  as  the  program  matures  through  the  defense  acquisition  framework. 
Specifically,  events  and  tasks  prior  to  Milestone  B  tend  to  address  T&E  integration  into 
the  process  less  sufficiently  than  those  after  Milestone  B.  Review  of  the  guidance 
governing  T&E  throughout  the  systems  acquisition  process  reveals  many  specific  areas 
of  interest.  These  areas  are  explored  in  greater  depth  in  the  following  paragraphs. 

1.  Requirements  Development 

As  described  in  Paragraph  II. D,  requirements  for  DoD  systems  are  detailed  in  a 
series  of  documents;  the  ICD,  the  CDD,  and  the  CPD.  These  documents  are  developed  in 
series  by  the  user  representative,  with  each  one  becoming  more  specific  as  the 
development  of  the  system  matures.  The  progression  of  requirements  documents  from 
ICD  to  CDD  to  CPD  is  shown  in  Figure  4. 
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FIGURE  4.  PROGRESSION  OF  REQUIREMENTS  DOCUMENTS  IN  THE  DEFENSE 

ACQUISITION  FRAMEWORK  [AFTER  6] 


After  the  user  representative  develops  these  requirements  documents,  the  program 
manager  is  responsible  for  translating  the  requirements  into  system  specifications  that 
define  the  design  of  the  system. 
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Guidance  governing  the  integration  of  T&E  into  the  development  of  the  ICD, 
CDD,  and  CDD  were  assessed  as  “Poor”  on  Tables  1,  3,  and  5,  respectively. 
Development  of  the  requirements  documents  that  ultimately  define  the  design  of  the 
system  is  driven  by  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS), 
which  is  governed  by  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170.01G 
[18].  The  instruction  makes  no  specific  reference  to  T&E.  Responsibilities  are  not  given 
below  the  service  level.  SECNAVINST  5000. 2D  further  defines  DoN’s  role  in  the 
requirements  development  process.  However,  regarding  T&E’s  participation  in  the 
requirements  development  process,  it  only  states  that  the  Chief  of  Naval  Operations 
(CNO)  Director  of  Test  and  Evaluation  and  Technology  Requirements  (N091)  is 
responsible  to  the  CNO  for  reviewing  the  capabilities  documents  for  testability  [9]. 

While  this  at  least  touches  on  T&E’s  role  in  the  requirements  development 
process,  it  is  not  sufficient.  In  addition  to  reviewing  for  testability,  the  T&E  community 
is  also  the  ideal  group  to  review  the  mission  relevancy  and  reasonableness  of  technical 
requirements.  Even  if  testability  were  the  only  concern,  CNO  (N091)  is  not  where  this 
responsibility  should  solely  reside.  Made-up  of  primarily  of  active-duty  service  members 
directly  from  the  fleet  with  little  to  no  formal  T&E  experience,  CNO  (N091)  is  not 
adequately  staffed  to  perform  this  significant  task.  Reviewing  the  testability  of  a 
technical  requirement  must  involve  the  end-state-tester  to  determine  if  the  technology, 
facilities,  and  techniques  exist  to  test  to  the  precision  necessary  to  evaluate  the  system’s 
achievement  of  the  requirement  and  if  it  can  be  done  within  reasonable  cost  and  schedule. 
While  CNO  (N091)  should  provide  an  oversight  role,  there  must  be  guidance  and  policy 
to  involve  the  responsible  test  organizations  in  the  review  of  requirements  documents  for 
testability,  mission  relevancy,  and  reasonableness. 

2.  Selection  of  Concept/System  to  be  Developed 

The  DoD  systems  acquisition  process  utilizes  many  processes  to  analyze  and 
decide  upon  the  system  to  be  developed.  These  processes  include  Analysis  of 
Alternatives  (AoA),  Request  for  Proposal  (RFP),  and  Source  Selection. 
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Guidance  governing  these  processes  was  assessed  as  “Poor”  in  Tables  1  and  2. 
Various  guidance  and  policy  documents  from  DoD,  DoN,  and  NAVAIR  govern  these 
processes.  From  the  federal  and  DoD  level,  these  guidance  and  policy  documents  include 
DoDI  5000.02,  and  the  Federal  Acquisition  Regulation  (FAR).  From  the  DoN  level,  they 
include  SECNAVINST  5000. 2D,  and  from  the  NAVAIR  level,  they  include  the 
NAVAIR  Acquisition  Guide  and  NAVAIRINST  4200.39B. 

No  T&E  related  guidance  is  provided  for  the  AoA.  Since  the  AoA  is  conducted 
without  direction  from  the  PM,  per  SECNAVINST  5000.2D,  without  any  T&E  related 
guidance,  there  is  little  to  no  chance  of  significant  input  from  the  DT  or  OT  community. 
Without  this  input,  T&E  related  impacts  of  choosing  a  particular  alternative  are  unlikely 
to  be  considered. 

RFP  and  Source  Selection  also  lack  sufficient  T&E  related  guidance,  with 
NAVAIRINST  4200.39B  making  no  mention  of  T&E  at  all.  SECNAVINST  5000.2D 
only  states  that  CNO  (N091)  shall  act  in  an  advisory  capacity. 

Guidance  for  these  processes,  especially  at  the  NAVAIR  level,  should  be  revised 
to  include  participation  of  the  T&E  discipline  (AIR-5.1  for  NAVAIR)  in  order  to  benefit 
from  the  insight  of  the  T&E  related  impacts  of  each  alternative  system. 

3.  SETR  Process  Participation 

NAVAIR  utilizes  the  Systems  Engineering  Technical  Review  (SETR)  process  as 
an  integral  part  of  the  systems  engineering  and  life  cycle  management  process  to  enable 
an  independent  assessment  of  emerging  designs  against  plans,  processes,  and  key 
knowledge  points  in  the  development  process  [23].  The  SETR  process  consists  of  a 
series  of  reviews  conducted  at  strategic  points  along  the  systems  acquisition  process. 
Figure  5  shows  the  timing  of  SETR  reviews  during  the  acquisition  process. 
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FIGURE  5.  TIMING  OF  SETR  REVIEWS  [FROM  23] 


Reviews  that  are  either  T&E  related  or  have  major  input  from  or  impact  on  T&E 
are  included  in  Tables  1-7.  Guidance  governing  these  reviews  was  assessed  as 
“Marginal”  to  “Good”  depending  on  the  specific  review.  Guidance  for  conducting  SETR 
reviews  for  NAVAIR  managed  programs  is  given  in  NAVAIRINST  4355. 19D. 

The  major  deficiency  noted  during  analysis  of  the  guidance  was  incomplete 
direction  regarding  review  participation  from  members  of  the  T&E  community.  The 
guidance  for  the  Initial  Technical  Review  (ITR),  Alternative  Systems  Review  (ASR), 
Preliminary  Design  Review  (PDR),  and  Critical  Design  Review  (CDR)  identifies  the 
Assistant  Program  Manager  for  Test  and  Evaluation  (APMT&E)  to  serve  on  the  technical 
review  board  to  ensure  that  all  T&E  requirements  are  addressed.  However,  no  further 
guidance  is  given  for  specific  T&E  participation.  The  guidance  for  the  System 
Requirements  Review  (SRR)  and  the  System  Functional  Review  (SFR)  does  not  even  go 
that  far,  stating  only  that  DT  and  OT  personnel  should  participate  in  the  reviews. 
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There  is  no  standardized  process  for  determining  who  (by  function  and  level  of 
authority)  should  participate  at  each  review  from  the  T&E  community.  As  a  result  of  this 
lack  in  guidance,  the  selection  of  T&E  personnel  to  participate  in  SETR  events  tends  to 
be  ad  hoc.  This  results  in  inconsistent  representation  of  the  T&E  competency  both  across 
programs  and  across  SETR  events  within  the  same  program. 

This  deficiency  does  not  primarily  reside  with  the  overarching  guidance  given  in 
NAVAIRINST  4355. 19C,  but  with  the  NAVAIR  T&E  community’s  implementation  of 
the  policy.  To  correct  this  inconsistency,  the  NAVAIR  T&E  community  should  develop 
a  standardized  procedure  for  determining  who  (by  function  and  authority  level)  should 
participate  at  each  SETR  event.  Flexibility  should  be  provided  in  this  standardized 
procedure  to  account  for  differences  in  program  type,  scope,  size,  and  visibility.  For 
example,  individual(s)  selected  for  a  radar  system  review  would  be  different  from  those 
chosen  for  an  aircrew  system  review.  Also,  different  individual(s)  would  be  appropriate 
for  an  ACAT  IV  review  as  compared  to  those  attending  a  high-visibility  ACAT  I  review. 

D.  RECOMMENDATIONS 

From  review  and  analysis  of  guidance  and  policy  governing  T&E’s  integration 
into  the  systems  acquisition  process,  as  described  in  the  preceding  paragraphs,  the 
following  recommendations  are  made: 

1.  Provide  guidance  and  policy  to  involve  the  responsible  test  organizations  in 
the  review  of  requirements  documents  for  testability,  mission  relevancy,  and 
reasonableness. 

2.  Revise  NAVAIR  guidance  related  to  AoA,  RFP,  and  Source  Selection,  to 
include  participation  of  the  T&E  discipline,  AIR-5.1. 

3.  Develop  NAVAIR  T&E  discipline  standardized  procedure  for  determining 
who  (by  function  and  authority  level)  should  participate  at  each  SETR  event. 
Flexibility  should  be  provided  in  this  standardized  procedure  to  account  for 
differences  in  program  type,  scope,  size,  and  visibility. 
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V.  ANALYSIS  OF  INTERVIEWS 


A.  INTRODUCTION 

In  order  to  better  understand  the  challenges  experienced  by  acquisition 
professionals  integrating  T&E  into  the  acquisition  process  for  Naval  Aviation,  interviews 
were  conducted  with  senior  leaders  and  subject  matter  experts.  The  interview  subjects 
had  widely  varied  experience  and  expertise  from  the  fields  of  program  management, 
systems  engineering,  and  T&E  across  multiple  programs  and  PEOs  within  NAVAIR. 
The  interview  subjects  were: 


•  CAPT  Richard  Muldoon 

Program  Manager,  PMA-261,  H-53  Heavy  Lift  Helicopters 

•  CAPT  Jeffrey  Penfield 

Program  Manager,  PMA-259,  Air-to-Air  Missile  Systems 

•  Mr.  Joseph  Wascavage 

Head,  Systems  Test  and  Experimentation  Management  Division 

•  Mr.  Robin  Locksley 

Assistant  Program  Executive  Officer  (APEO)  for  Test  and  Evaluation, 
PEO(Unmanned  Aviation  and  Strike  Weapons) 

•  Mr.  Michael  Gomes 

APEO  for  Engineering,  PEO(Air  ASW,  Assault,  and  Special  Mission 
Programs) 

•  Mr.  Gary  Evans 

APEO  for  Engineering,  PEO(Unmanned  Aviation  and  Strike  Weapons) 

•  Mr.  Neal  Siegel 

Head,  T&E  Processes,  Standards,  and  Special  Programs  Office 

•  Mr.  David  Roberts 

Chief  Test  Engineer,  Atlantic  Test  Range 

•  Mr.  Christian  Rice 

Chief  Test  Engineer,  HX-21  Rotary  Wing  Test  Squadron 
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•  Mr.  James  Schmidt 

APM(T&E),  PMA-261,  H-53  Heavy  Lift  Helicopters 

The  interview  questions  used  to  guide  the  interviews  are  provided  in  the 
Appendix. 

B.  ANALYSIS  RESULTS 

The  interview  responses  were  reviewed  and  analyzed  to  look  for  common  themes. 
While  the  interview  responses  were  very  wide-reaching,  with  each  interview  subject 
naturally  tending  to  focus  on  their  particular  areas  of  expertise  and  experience,  several 
common  themes  emerged  regarding  areas  for  improvement  integrating  T&E  into  the 
acquisition  process  for  Naval  Aviation.  These  areas  are  discussed  in  greater  depth  in  the 
following  paragraphs. 

1.  Development  of  Test  Program  Estimates 

One  area  of  weakness  commonly  cited  by  the  interview  subjects  was  developing 
cost  and  schedule  estimates  for  T&E  programs.  These  estimates,  done  very  early  in  the 
system’s  development,  form  the  foundation  of  the  T&E  program,  as  they  establish  the 
funding  and  schedule  profile  allocated  to  the  test  program.  However,  the  interviews 
identified  a  major  shortcoming  in  the  test  program  estimation  process. 

Although  flight  test  is  relatively  mature  as  a  process,  having  been  done  in  a 
relatively  systematic  manner  since  the  establishment  of  formal  test  pilot  training 
programs  in  the  late  1940s,  flight  test  execution  data  has  not  been  captured  and  recorded 
in  a  manner  that  easily  aids  estimation  and  planning  for  future  flight  test  programs. 

There  is  currently  no  tool  that  captures  test  efficiency  and  test  throughput  of  past 
test  programs.  Test  efficiency  and  throughput  are  significant  factors  impacting  the  ability 
to  hold  to  a  test  schedule.  Major  factors  that  impact  test  efficiency  and  throughput 
include  aircraft  availability  due  to  maintenance,  weather,  range  scheduling,  and  the 
necessity  to  re-fly  test  points  due  to  test  complexity. 
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If  a  computer-aided  tool  or  database  captured  the  results  of  prior  flight  test 
programs,  it  would  be  of  great  use  for  planning  future  test  programs.  An  example  of 
information  that  could  be  derived  from  such  a  tool  might  be  that  for  aircraft  platform  ‘W’ 
operating  out  of  flight  test  location  ‘X’  performing  ‘Y’  type  of  testing,  the  average 
throughput  is  ‘Z’  test  sorties  per  month.  While  no  two  test  programs  are  exactly  alike, 
this  infonnation  would  provide  a  base  of  knowledge  to  make  realistic  estimates  for 
planning  future  test  programs. 

A  lot  of  this  historical  information  is  already  partially  captured  in  the  minds  of 
those  who  experienced  the  test  programs.  While  this  corporate  knowledge  is  very 
valuable,  it  has  two  significant  shortcomings. 

First,  because  the  information  exists  only  in  the  minds  of  those  who  experienced 
it,  it  is  only  valuable  if  they  personally  participate  in  and  influence  planning  for  the  next 
test  program.  If  they  either  are  not  personally  involved  in  the  test  planning  or  if  those 
who  are  personally  involved  aren’t  aware  of  the  experienced  persons’  knowledge  and  do 
not  seek  it  out,  it  is  lost  and  is  of  no  value  to  estimating  the  schedule  and  resources  for  the 
next  test  program. 

Second,  because  the  knowledge  is  often  anecdotal  in  nature,  it  may  not  be 
effective  in  justifying  and  defending  test  program  estimates  when  they  are  challenged. 
Due  to  tight  fiscal  constraints,  there  is  tremendous  pressure  to  make  program  schedules 
and  budgets  success-oriented.  This  leads  to  pressure  to  develop  test  schedules  and 
funding  profiles  that  assume  success  and  high  levels  of  efficiency  and  throughput.  It  is 
the  test  team’s  role  to  challenge  this  pressure  by  pushing  for  schedules  and  resources  that 
are  realistic,  based  on  historical  test  efficiency  and  throughput.  However,  if  this 
information  is  only  anecdotal  in  nature,  it  is  challenging  to  justify  adding  time  and 
resources  to  achieve  a  realistic  plan.  If  a  recognized  tool  existed  to  formally  record  and 
present  this  information,  justification  of  T&E  schedules  and  budgets  based  on  ‘realistic’ 
assumptions  would  be  greatly  aided.  It  is  much  easier  to  defend  planning  assumptions 
made  based  on  documented  historical  data  rather  than  anecdotal  knowledge. 
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Development  of  such  a  test  program  planning  tool  would  require  significant  effort 
to  develop  and  maintain;  however,  the  investment  would  yield  great  returns  in  the  form 
of  realistic,  justifiable  and  defendable  estimates  of  test  schedules  and  budgets.  Data  to 
support  development  of  a  tool  should  be  available  in  DT&E  and  OT&E  reports  from 
previous  programs.  Inputs  from  both  Navy  and  Air  Force  test  programs  could  contribute 
to  the  database. 

2.  Education  of  T&E  Workforce 

Another  area  frequently  cited  in  the  interview  responses  was  the  need  for 
education  of  the  T&E  workforce.  Although  the  Naval  Aviation  T&E  workforce  as  a 
whole  is  seen  as  highly  educated  and  well  trained,  there  are  certain  knowledge  areas  that 
were  cited  as  weak  and  in  need  of  improvement. 

a.  T&E  Schedule  and  Cost  Estimation  Training 

One  of  those  areas  is  inadequate  training  for  developing  T&E  schedules 
and  cost  estimates.  As  noted,  T&E  estimates  performed  very  early  in  the  T&E  planning 
process  lay  the  foundation  for  the  test.  Although  foundational,  this  area  of  T&E  planning 
may  be  the  weakest  area  of  knowledge  and  skill  within  the  T&E  workforce. 

As  will  be  further  illustrated  in  the  next  section,  DT&E  practitioners  are 
given  extensive  training  on  detailed  test  planning,  test  execution,  and  test  reporting. 
However,  no  formal  training  is  provided  on  the  skill  of  long-term  T&E  schedule  and  cost 
estimation.  Combined  with  a  lack  of  formal  tools  for  estimation  as  described  in  the 
section  above,  the  resulting  estimates  tend  to  be  of  widely  varying  degrees  of  quality, 
primarily  based  on  infonnal  rules  of  thumb,  rather  than  a  rigorous,  systematic  process. 

This  problem  is  compounded  by  the  attitudes  and  values  of  those  who  are 
tasked  to  perfonn  T&E  estimates,  mainly  testers.  T&E,  particularly  Naval  Aviation 
flight  test,  tends  naturally  to  attract  people  whose  interests  and  skills  are  focused  on  the 
hands-on,  near-term  tasks  of  preparing  for  and  executing  flight  test  operations.  They  are 
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often  not  interested  in  or  formally  trained  in  performing  long-tenn  cost  and  schedule 
estimates.  As  a  result,  this  aspect  of  the  job  tends  to  get  little  attention  compared  to 
execution  of  flight  test  operations. 

To  overcome  these  challenges,  formal  training  in  T&E  schedule  and  cost 
estimation  is  necessary  to  give  the  T&E  workforce  both  the  necessary  skills  to  perform 
high-quality  estimates  and  an  appreciation  for  the  impact  early  estimating  has  on  the 
success  of  the  overall  T&E  program. 

b.  Training  of  OT  OTDs 

Another  area  of  weakness,  most  often  cited  by  interview  subjects  with 
experience  working  within  the  field  of  OT,  is  insufficient  training  for  OT  personnel, 
particularly  Commander  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 
Operational  Test  Directors  (OTDs). 

OTDs  are  responsible  for  conducting  detailed  OT  planning,  supervision  of 
test  execution,  and  documentation  of  test  results  [26].  OTDs  are  typically  0-3/0-4  level 
officers  (Lieutenants  and  Lieutenant  Commanders)  pulled  directly  from  the  fleet  with  no 
prior  T&E  experience.  They  are  given  tremendous  responsibility,  considering  that  the 
tests  that  they  plan  and  the  reports  that  they  write  will  be  seen  as  the  final  report  card  for 
an  acquisition  program,  recommending  whether  or  not  a  system  should  be  fielded.  These 
tests  and  test  reports  have  very  high  visibility,  having  great  political  and  public  relations 
impact  for  the  program  under  test. 

However,  the  training  given  to  OTDs  to  prepare  them  for  such  a  high 

responsibility  is  insufficient.  New  OTDs  reporting  to  COMOPTEVFOR  are  given  a 

three-day  course,  “To  provide  OTD  and  support  personnel  with  a  baseline  knowledge  of 

weapon  system  acquisition,  and  introduce  them  to  policies,  procedures,  documentation 

and  reports  required  by  DOD  and  SECNAV  in  conducting  OT&E  [33].”  While  the 

course  does  an  admirable  job  presenting  the  material  as  well  as  possible,  given  the  time 

constraints,  three  days  is  simply  not  enough  to  cover  the  material  in  sufficient  depth. 

This  is  especially  true  given  that  the  typical  OTD  has  no  prior  T&E  or  acquisition-related 

experience  and  is  given  such  a  high-level  of  responsibility  to  properly  perform  the  OT&E 
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function  of  the  systems  acquisition  process.  As  a  result,  the  vast  majority  of  learning 
how  to  plan,  conduct  and  report  on  OT  is  accomplished  “on-the-job”  getting  unofficial 
anecdotal  guidance  from  peers  [33]. 

The  degree  of  insufficiency  a  three-day  course  provides  is  made  even 
more  evident  when  contrasted  with  the  level  of  fonnal  training  given  to  the  OTD’s 
counterpart  in  the  DT&E  world,  the  DT  project  officer.  Nearly  all  Naval  Aviation  DT 
project  officers  are  sent  through  the  United  States  Naval  Test  Pilot  School  (USNTPS)  or 
another  service  equivalent  such  as  the  United  States  Air  Force  Test  Pilot  School.  The 
USNTPS  curriculum  is  48  weeks  long,  proving  instruction  in  “academics,  flight  test 
preparation,  flight  test  conduct,  data  collection,  data  reduction,  and  test  report  preparation 
[44].”  While  the  USNTPS  and  OTD  Course  curricula  do  differ  significantly  in  areas  of 
focus,  the  disparity  between  a  48-week  course  for  DT  project  officers  and  a  3-day  course 
for  OTDs  clearly  shows  a  disconnect  in  the  value  that  is  being  placed  on  their 
professional  educations.  This  is  especially  troubling  given  the  tremendous  impact  the  OT 
report  has  on  the  success  or  failure  of  an  acquisition  program. 

The  inadequate  training  of  OTDs  often  forces  them  to  rely  on  their  DT 
counterparts  or  development  contractors  for  support  in  planning  and  analysis  of  OT&E. 
While  this  can  be  helpful,  the  DT  testers  and  development  contractors  have  different 
objectives  and  their  views  are  sometime  in  conflict  with  OT&E  policies.  OT&E,  by  law 
and  policy,  must  be  performed  independently  from  and  uninfluenced  by  development 
organizations. 


3.  Selection  of  Test  Facilities 

One  of  the  most  important  decisions  made  early  in  the  test  planning  process  is 
selection  of  the  test  facilities  to  be  used.  For  Naval  Aviation,  this  includes  the  selection 
of  laboratories  and  engineering  centers  for  component  level  testing  as  well  as  flight  test 
centers  and  major  ranges  for  system  level  flight  testing.  Selection  of  these  test  facilities 
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has  a  major  impact  on  the  cost,  schedule,  and  performance  of  the  test  program.  As 
significant  as  this  decision  is,  the  interviews  revealed  areas  of  weakness  impacting  this 
decision-making  process. 

a.  Knowledge  of  Test  Facility  Capabilities 

There  currently  exists  no  single  source  for  detailed  information  regarding 
test  facility  capabilities.  Across  each  of  the  services,  DoD  as  a  whole,  other  federal 
government  organizations,  and  private  industry,  there  exists  a  wealth  of  test  capability. 
However  without  a  common  source  of  test  capability  information,  test  teams  are 
generally  dependent  upon  the  “tribal  knowledge”  within  the  team  regarding  the 
capabilities  of  test  facilities.  As  a  result,  many  appropriate  facilities,  within  both  the 
government  and  private  industry  are  never  considered  unless  the  test  planning  team  had 
prior  experience  or  knowledge  of  testing  at  those  facilities. 

A  single  searchable  repository  of  test  facility  capabilities  would  greatly  aid 
the  ability  of  the  test  planner  to  choose  the  test  facility  that  best  meets  the  technical, 
schedule,  and  fiscal  requirements  of  the  test  program.  An  example  of  how  such  a 
repository  might  be  used  is  if  a  test  planner  was  planning  for  a  test  that  required  the 
measurement  of  an  aircraft’s  or  aircraft  component’s  radar  cross  section,  he  could  do  a 
search  of  the  tool  for  “radar  cross  section  measurement”  and  find  a  list  of  all  facilities, 
both  government  and  private  industry  that  have  the  capability  for  radar  cross  section 
testing.  The  proposed  tool  should  list  at  a  minimum  a  short  description  of  the  facility’s 
capabilities  in  that  field  along  with  a  point  of  contact  for  further  information. 

Development  of  such  a  tool  would  require  participation  from  test  facilities 
to  provide  the  necessary  information  and  periodic  updates.  However,  the  test  facility 
would  be  motivated  to  provide  this  information  as  it  would  advertise  their  capability  and 
promote  the  use  of  their  facility.  The  “owner”  of  this  tool  could  be  either  at  the  NAVAIR 
T&E  competency  level  or  at  a  higher  DoN  or  DoD  organization  to  promote  its  use 
throughout  these  higher  levels.  At  the  DoD  level,  a  logical  organization  to  own  this  tool 
is  the  Test  Resource  Management  Center  (TRMC)  within  USD(AT&L). 
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b.  Selection  of  Contractor  vice  DoD  T&E  Facilities 

An  area  of  guidance  weakness  related  to  test  facilities  identified  by  the 
interviews  is  the  decision  to  use  DoD  or  contractor  test  facilities  for  a  test  program. 
Currently,  the  selection  of  DoD  or  contractor  test  facilities  rests  solely  with  the  individual 
program.  No  guidance  currently  exists  that  requires  that  the  impact  of  test  facility 
selection  (DoD  or  contractor)  on  DoD  as  a  whole  be  assessed.  For  example,  while  the 
choice  of  using  the  prime  contractor’s  test  facility  over  a  DoD  operated  test  facility  may 
be  advantageous  to  an  individual  program  from  a  cost  and  schedule  perspective,  it  may 
have  a  negative  overall  impact  on  DoD. 

As  the  number  of  new  DoD  programs  continues  to  decrease,  the  impact  of 
any  individual  program  choosing  not  to  use  DoD  test  facilities  has  a  greater  impact  on  the 
health  of  those  facilities.  Choosing  contractor  facilities  over  govermnent  test  facilities 
not  only  has  a  direct  financial  impact,  but  more  importantly  has  a  long-term  impact  from 
lost  opportunity  to  sustain/develop  DoD-held  expertise  and  experience  in  core  T&E 
disciplines. 

Policy  and  guidance  should  be  developed  to  require  this  assessment  be 
conducted  as  a  part  of  the  test  facility  selection  process.  Choosing  to  use  a  contractor  test 
facility  instead  of  a  government  test  facility  is  not  inherently  bad  and  in  many  cases  may 
be  the  best  decision  for  DoD,  however  the  short-tenn  gains  for  the  individual  program 
must  be  weighed  against  the  impact  to  NAVAIR,  DoN,  and  DoD  test  capabilities  as  a 
whole. 


4.  T raceability  of  Test  Requirements 

An  identified  area  of  needed  improvement  by  the  interview  subjects  was 
traceability  of  test  requirements.  Adequate  traceability  of  test  events  to  technical 
requirements  is  key  to  a  successful  T&E  program.  Test  requirements  traceability  ensures 
that  sufficient  test  data  is  generated  to  allow  evaluation  of  the  system’s  technical 
requirements.  Traceability  also  serves  to  justify  the  scope  of  a  test  program, 
documenting  that  the  right  amount  of  testing  is  being  accomplished,  without  “gold- 
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plating”  the  test  requirements  or  succumbing  to  the  natural  engineering  perspective  of 
“more  data  is  better.”  This  allows  the  test  team  to  defend  the  test  program  when  cost  and 
schedule  are  “in  the  cross-hairs”  for  programmatic  cuts. 

Within  Naval  Aviation,  traceability  of  test  requirements  is  currently  left  up  to 
individual  programs  to  determine  the  manner  in  which  it  is  to  be  done.  As  a  result, 
traceability  of  test  requirements  is  done  in  an  ad  hoc  fashion  among  T&E  programs. 
While  larger  major  acquisition  programs  often  use  formal  tools  such  as  IBM’s  DOORS® 
software,  there  is  no  standardized  tool  or  format  for  requirements  traceability. 

A  prescriptive  policy  is  needed  to  provide  guidance  on  T&E  requirements 
traceability  across  Naval  Aviation  T&E  programs.  An  added  benefit  is  that  a 
standardized  policy  would  also  aid  the  schedule  and  cost  estimating  processes  discussed 
earlier  by  providing  input  to  the  T&E  estimator  regarding  test  data  required,  which 
directly  drives  schedule  and  funding  required. 

5.  Developmental  Test  Reporting  Process 

The  final  area  that  was  commonly  cited  as  an  area  where  improvement  is 
necessary  was  the  developmental  test  reporting  process.  The  purpose  of  the  test  report  is 
to  provide  feedback  to  the  developer  with  decision-level  quality  data.  The  shortcoming 
commonly  noted  regarding  NAV AIR’s  test  reporting  process  was  not  regarding  the 
quality  of  the  report,  but  the  timeliness  of  the  report. 

Review  of  the  applicable  guidance  illustrates  a  disparity  between  OT  reporting 
requirements  and  DT  reporting  requirements.  SECNAVINST  5000.2D  requires  that, 

COMOPTEVFOR  shall  issue  operational  test  reports  for  AC  AT  I  and  I A 

programs  within  90  days  following  completion  of  testing.  All  other 

operational  test  reports  are  due  within  60  days  of  test  completion.  [9] 

This  guidance  is  flowed  down  and  repeated  in  the  COMOPTEVFOR  OTD 
Manual  [26].  No  such  timeline  guidance  is  given  for  developmental  test.  SECNAVINST 
5000. 2D  only  states  that, 
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A  report  of  results  for  all  DT&E  conducted  in  DoN  shall  be  provided  to 

the  appropriate  decision  authority  and  to  the  OTA  as  needed.  [9] 

Accordingly,  the  guidance  documents  that  govern  NAVAIR  DT&E  report  writing 
are  silent  regarding  timelines  for  issuing  test  reports  following  completion  of  testing  [30, 
31].  Naturally,  this  leads  to  widely  disparate  results  regarding  timely  report  issuance. 
These  disparate  results  are  not  only  due  to  widely  varied  times  from  end  of  test  to 
completion  of  the  draft  report,  but  also  widely  varied  times  to  completion  of  the  test 
report  approval  process. 

Unfortunately,  this  often  results  in  the  completed  reports  being  delivered  to  the 
decision  authority  after  the  necessary  decisions  which  the  report  supports  are  required  to 
be  made.  For  example,  readiness  for  OT&E  cannot  be  assessed  adequately  without  a 
DT&E  report,  thus  the  OTRR  decision  is  impacted  by  late  delivery  of  a  DT&E  report. 
This  forces  the  decision  maker  to  either  delay  making  decisions  or,  more  likely,  make 
decisions  based  on  informal  or  incomplete  data.  The  current  processes  are  often  failing 
to  deliver  decision-level  quality  data  at  the  time  the  decision  needs  to  be  made.  To 
correct  this,  policy  should  be  provided  to  mandate  reporting  timelines  for  DT  test  reports, 
holding  both  the  report  provider  and  reviewing  authorities  accountable  to  deliver  test 
reports  in  a  timely  manner. 

C.  RECOMMENDATIONS 

From  review  and  analysis  of  the  interview  responses  given  in  the  preceding 
paragraphs,  the  following  recommendations  are  made: 

1.  Develop  a  tool  to  capture,  record,  and  present  historical  flight  test  data  to 
provide  the  basis  for  realistic  assumptions  of  flight  test  efficiency  and 
throughput  for  developing  flight  test  schedule  and  budget  estimations. 

2.  Provide  formal  training  on  T&E  schedule  and  cost  estimation  to  T&E 
personnel  involved  in  performing  these  tasks. 
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3.  Provide  additional  fonnal  training  to  OTDs  and  support  personnel  to  prepare 
them  for  the  tasks  of  OT  planning,  supervision  of  test  execution,  and 
documentation  of  test  results. 

4.  Develop  a  tool  to  provide  a  single  repository  of  test  facility  capabilities  to  aid 
the  selection  of  test  facilities  that  best  meet  the  technical,  fiscal,  and  schedule 
requirements  of  the  test  program. 

5.  Develop  policy  and  guidance  to  assess  the  impact  of  choosing  contractor  vice 
government  test  facilities  on  NAVAIR,  DoN,  and  DoD  as  a  whole  to  aid  the 
selection  of  test  facilities. 

6.  Develop  policy  and  guidance  to  standardize  traceability  of  test  requirements 
across  Naval  Aviation  T&E  programs. 

7.  Develop  policy  and  guidance  for  DT&E  reporting  timelines  to  hold  both  the 
report  provider  and  approval  authorities  accountable  for  timely  delivery  of  test 
reports. 
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VI.  CONCLUSION  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

T&E  is  incorporated  throughout  both  the  systems  engineering  and  DoD  system 
acquisition  processes.  T&E  is  the  mechanism  for  accomplishing  verification  in  the 
systems  engineering  process  and  characterizing  technical  risk  of  achieving  a  proper  final 
design  solution.  T&E  is  a  critical  and  continuous  activity  throughout  the  DoD  systems 
acquisition  process  to  ensure  that  cost,  schedule,  and  perfonnance  requirements  are 
satisfied  with  acceptable  levels  of  risk. 

At  each  stage  of  the  process,  T&E  confirms  whether  people,  product,  and  process 
solutions  meet  or  exceed  the  user’s  requirements.  During  early  phases  of  the  DoD 
acquisition  process,  T&E  activities  may  take  the  form  of  analysis,  modeling,  simulation, 
and  proof  of  concept  tests  for  system,  subsystem,  and  component  levels.  Later  stages 
will  focus  more  on  examination,  demonstration,  and  testing  to  verify  the  function  of  the 
design  and  validate  the  produced  system  meets  user  requirements. 

Guidance  for  integration  of  T&E  into  the  systems  acquisition  process  for  Naval 
Aviation  flows  from  Federal  law,  to  DoD,  to  DoN,  to  NAVAIR  for  implementation 
through  NAVAIR  policy  and  guidance.  Through  analysis  of  this  guidance  along  with 
interviews  of  senior  acquisition  professionals,  the  following  areas  of  weakness  were 
discovered  regarding  integrating  T&E  into  the  systems  acquisition  process: 

1 .  Requirements  Development 

2.  Selection  of  Concept/System  to  be  Developed 

3.  SETR  Process  Participation 

4.  Development  of  Test  Program  Estimates 

5.  Education  of  the  T&E  Workforce 

6.  Selection  of  Test  Facilities 
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7.  Traceability  of  Test  Requirements 

8.  Developmental  Test  Reporting  Requirements 

B.  RECOMMENDATIONS 

Through  analysis  of  T&E  related  guidance  and  interviews  of  senior  acquisition 
professionals,  the  following  recommendations  are  made  to  address  the  areas  of  weakness 
cited  above: 

1.  Provide  guidance  and  policy  to  involve  the  responsible  test  organizations  in 
the  review  of  requirements  documents  for  testability,  mission  relevancy,  and 
reasonableness. 

2.  Revise  NAVAIR  guidance  related  to  AoA,  RFP,  and  Source  Selection,  to 
include  participation  of  the  T&E  discipline,  AIR-5.1. 

3.  Develop  NAVAIR  T&E  discipline  standardized  procedures  for  determining 
who  (by  function  and  authority  level)  should  participate  at  each  SETR  event. 
Flexibility  should  be  provided  in  this  standardized  procedure  to  account  for 
differences  in  program  type,  scope,  size,  and  visibility. 

4.  Develop  a  tool  to  capture,  record,  and  present  historical  flight  test  data  to 
provide  the  basis  for  realistic  assumptions  of  flight  test  efficiency  and 
throughput  for  developing  flight  test  schedule  and  budget  estimations. 

5.  Provide  formal  training  on  T&E  schedule  and  cost  estimation  to  T&E 
personnel  involved  in  performing  these  tasks. 

6.  Provide  additional  fonnal  training  to  OTDs  and  support  personnel  to  prepare 
them  for  the  tasks  of  OT  planning,  supervision  of  test  execution,  and 
documentation  of  test  results. 

7.  Develop  a  tool  to  provide  a  single  repository  of  test  facility  capabilities  to  aid 
the  selection  of  test  facilities  that  best  meet  the  technical,  fiscal,  and  schedule 
requirements  of  the  test  program. 
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8.  Develop  policy  and  guidance  to  assess  the  impact  of  choosing  contractor  vice 
government  test  facilities  on  NAVAIR,  DoN,  and  DoD  as  a  whole  to  aid  the 
selection  of  test  facilities. 

9.  Develop  policy  and  guidance  to  standardize  traceability  of  test  requirements 
across  Naval  Aviation  T&E  programs. 

10.  Develop  policy  and  guidance  for  DT&E  reporting  timelines  to  hold  both  the 
report  provider  and  approval  authorities  accountable  for  timely  delivery  of  test 
reports. 
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APPENDIX:  INTERVIEW  QUESTIONS 


The  following  questions  were  used  to  guide  each  interview: 


1.  In  what  capacities  have  you  been  involved  with  T&E  during  the 
acquisition  process? 

2.  Do  you  believe  that  current  policies  and  guidance  do  a  satisfactory 
job  of  integrating  T&E  into  the  acquisition  process? 

3 .  What  areas  of  weakness  do  you  see  in  current  guidance  governing 
T&E? 

4.  Are  there  any  areas  of  T&E  guidance  that  you  have  seen  routinely 
not  followed? 

5.  What  challenges  have  you  experienced  integrating  T&E  into  the 
acquisition  process? 

6.  What  improvements  can  be  made  to  policies,  procedures,  and 
guidance  to  better  integrate  T&E  into  the  systems  acquisition 
process? 
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